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Reducing Design Risk Using Robust Design Methods: 
A Dual Response Surface Approach 

Resit Unal 

Old Dominion University 

Ozgur Yeniay 
Hacettepe University 


1. Introduction 

Space transportation system conceptual design is a multidisciplinary process containing 
considerable element of risk. Risk here is defined as the variability in the estimated 
(output) performance characteristic of interest resulting from the uncertainties in the 
values of several disciplinary design and/or operational parameters. Uncertainties from 
one discipline (and/or subsystem) may propagate to another, through linking parameters 
and the final system output may have a significant accumulation of risk. This variability 
can result in significant deviations from the expected performance. Therefore, an 
estimate of variability (which is called design risk in this study) together with the 
expected performance characteristic value (e.g. mean empty weight) is necessary for 
multidisciplinary optimization for a robust design. Robust design in this study is defined 
as a solution that minimizes variability subject to a constraint on mean performance 
characteristics. 

Even though multidisciplinary design optimization has gained wide attention and 
applications, the treatment of uncertainties to quantify and analyze design risk has 
received little attention. This research effort explores the “dual response surface” 
approach to quantify variability (risk) in critical performance characteristics (such as 
weight) during conceptual design. 

2. Multidisciplinary Conceptual Design 

In launch vehicle conceptual design studies, system performance can generally be 
determined by the use of computerized analysis tools available in many disciplines. 
However, these complex sizing and performance evaluation computer codes utilize iterative 
algorithms. In many cases, they are expensive and difficult to integrate and use directly for 
multidisciplinary design optimization (MDO). An alternative is to construct mathematical 
models that approximate the relationships between performance characteristics and design 
variables. These approximation models, called response surface models, may then used to 
integrate the disciplines using mathematical programming methods and for 
multidisciplinary system level optimization. 

3. Response Surface Methods 
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A second-order approximation model of the form given below (Equation 1) is commonly 
used in approximation model building since in many cases it may adequately model the 
response surface especially if the region of interest is sufficiently limited. 

Y = b 0 + ? bi x| + ? bjj Xj 2 + ? ? bjj xj xj (1) 

In equation 1, the xj terms are the input variables that influence the response Y, and b 0 , 
bj, and bij are estimated regression coefficients. The cross terms represent two-parameter 
interactions, and the square terms represent second-order non-linearity. There are various 
response surface techniques that may be utilized to sample the design space efficiently to 
construct a second order approximation model. Some of these are, central composite 
designs (Montgomery, 1991; Myers, 1971; Khuri and Cornell, 1987) and D-Optimal 
designs (Craig, 1978; Roux et. al.,1996). Response surface methods using these designs 
have been applied to various multidisciplinary design optimization problems (Guinta et. 
Al, 1996; Lepsch et al, 1995; Unal et. al, 1996; Roux et al, 1996; Unal et al, 1998). The 
main advantage is that response surface methods can aid multidisciplinary design 
integration, and provide rapid design analysis and optimization capability in many 
applications. However, in these (and most) response surface model building applications, 
only the mean value for the performance characteristic (Y) was studied and optimized 
ignoring design risk or variability due to uncertain design parameter values (noise). 

4. Dual Response Surfaces 

Response surface methodology is designed to construct an approximation model for the 
response Y. This approximation model is then used to determine the best values of 
design parameters that optimize the response or the critical quality characteristic. In such 
problems typically the focus is on the mean value of Y where variance “S 2 ” is assumed to 
be small and constant. However, if the variance is not constant, only constructing a 
response surface model for the mean may not be adequate and optimization results can be 
misleading. Kim and On (1998) note that the assumption of constant variance is often 
not practically valid and in these events, classical response surface methods can be 
misleading. 

An alternative to classical response surface methods is the “dual response surface” 
approach, which builds two models; one for the mean and one for the standard deviation 
(Vining & Myers, 1990; Lin & Tu, 1995; Wang, 1996; Del Castillo and Montgomery, 
1993; Del Castillo, Fan & Semple, 1997; Tong, 2001, Tang and Xu, 2002). 

Yu = b 0 + ? bi xj + ? bjj xj 2 + ? ? b-j X j X j + e (2) 

Y a = c 0 + ? ci xj + ? qj xj 2 + ? ? qj Xj xj + e (3) 

The dual response surface approach simultaneously considers the estimated mean (p) and 
the standard deviation (a) as a measure of design risk. 

The steps in the dual response surface approach can be summarized as: 
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1) Conduct the experiments to sample the design space using a response surface 
technique such as central composite design or a D-Optimal design, 

2) Repeat the experiments over the range of uncertain design (noise) parameters to 
capture variability, 

3) Compute the mean and the variance for each experiment 

4) Construct response surfaces for both the mean and variance, 

5 ) Determine the best values of xj to optimize the mean response subject to a constraint 

on the variance or determine the best values of xj to minimize the variance subject to 
a constraint on mean response. 

The dual response surface approach therefore can be utilized in approximation model 
building to quantify the variability in the mean value of the performance characteristic 
and for optimization considering both the mean response and variability to reduce design 
risk. Several studies successfully utilized the Dual Response methodology to 
optimization studies that considered the mean and the variability (Luner, 1994; Copeland 
& Nelson, 1996; Del Castillo, Fan & Semple, 1999, Dhavlikar, Kulkami & Mariappan, 
2002 ). 

In this study, the dual response surface approach is applied to the conceptual design of a 
wing-body vehicle, configuration study. The results are presented and the strengths and 
limitations of the dual response surface approach are discussed. 

5. Wing Body Launch Vehicle Configuration Study 

An application of dual response surfaces is described for a configuration optimization 
study of a rocket-powered, single-stage-to-orbit launch vehicle. The vehicle has been 
sized by the Vehicle Analysis Branch (VAB) engineers to perform a 25,000 lb. payload 
delivery to the International Space Station from Kennedy Space Center. Near term 
structures and subsystem technologies are assumed in its design. The vehicle has a wing- 
body configuration with a slender, round cross-section fuselage and a clipped delta wing. 
The delta wing has elevon control surfaces for aerodynamic roll and pitch control. Small 
vertical fins, called tip fins, are located at the wing tips for directional control and a body 
flap extends rearward from the lower base of the fuselage to provide additional pitch 
control. 

The purpose in this study is to determine the best values of the design parameters (that 
satisfy aerodynamic constraints) to minimize mean empty weight (WT) subject to a 
constraint on the variance. Therefore, empty weight is the critical performance 
characteristics to be optimized in this case. This multidisciplinary optimization and 
sensitivity study involved the following steps. 

5.1. Identify Design Variables and Feasible Ranges 

Four weight & sizing and aerodynamics design parameters were varied over a fixed 
range. The four common parameters included in aerodynamics and weights & sizing 
analysis were the fineness ratio (defined as the fuselage length divided by diameter), the 
wing area, the tip fin area, and the body flap area. The other parameters included were 
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ballast-weight and mass-ratio for weights and sizing analysis and angle of attack and 
elevon deflection for aerodynamics analysis. 

In this study, ballast-weight and mass-ratio were taken as the noise variables. Noise 
variables are those that one has little or no control on the values during operation. 
Therefore, one design goal is to reduce the variability in performance (weight in this case) 
due to the uncertain values of these parameters. 


The ranges for the six weights & sizing parameters are given at Table 1. 
Design Parameters Rang e 


Fineness ratio (FR) 

Wing area ratio (WA) 
Tip fin area ratio (TP) 
Body flap area ratio (BF) 

Noise Parameters 


Ballast weight (BL) 
Mass ratio (MR) 


4 7 

10 20 

0.5 3 

0 1 


0 0.04 

7.75 8.25 


Table 1. Weights & Sizing Parameters and Ranges 


5.2. Construct the Experimental Design Matrix 

The next step is to construct an experimental design matrix that can enable the 
construction of a second-order response model for 4 parameters. Carpenter (1993) has 
conducted a study comparing the performance of various experimental design methods 
for approximation model building in terms of the quality of fit in the region studied and in 
terms of the number of design points required. His findings su gge st that the D-optimality 
criteria as a good approach for constructing experimental designs for computer 
experiments. Other studies in the literature also found that the D-optimality criterion 
provides a rational means for creating experimental designs for an irregularly shaped 
response surfaces (Guinta et. al, 1996). Therefore, D-optimal designs are used in this 
study to construct an experimental design to sample the design space. The following 
section gives a background on D-optimal designs. 

5.3. D-Optimal Designs 

A statistical measure of goodness of a model obtained by least squares regression analysis 
is the minimum generalized variance of the estimates of the model coefficients. One way 
to construct a quadratic model using minimum point designs, leading to minimized 
variance of the least squares estimates, is to use the D-optimality criterion. Consider the 
problem of estimating the coefficients of a linear approximation model below by least 
squares regression analysis. 
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y = b 0 + ? bi xj (4) 

Equation 4 can be expressed in matrix notation as: 

Y = XB + e (5) 

Where Y is a vector of observations, e is the vector of errors, X is the design matrix and 
B is a vector of unknown model coefficients (b 0 and bi). The design matrix is a set of 

combinations of the values of the coded variables, which specifies the settings of the 
design parameters to be performed during experimentation. B can be estimated by using 
the least squares method as: 

B = (XX)' 1 XY (6) 

A measure of accuracy of the column of estimators, B, is the variance-covariance matrix 
which is defined as; 

V(B) = a 2 (XX)' 1 (7) 

where a 2 is the variance of the error. The V(B) matrix is a statistical measure of the 
goodness of the fit. Equation 7 indicates that V(B) is a function of (XX)" 1 and therefore, 
one would want to minimize (XX)' 1 to improve the quality of the fit. Statisticians have 

shown that minimizing (XX)' 1 is equivalent to maximizing the determinant of XX 
(Mitchell, 1974, Montgomery, 1991, Craig, 1978, Unal et al, 1996). Therefore, 
generating a design matrix which enables the construction of a good least squares 
approximation model translates to maximizing the determinant of the XX matrix and 
experimental designs that maximize |XX| are referred to as D-optimal designs. Here, “D” 
stands for the determinant of the XX matrix associated with the model. 

A number of authors have developed algorithms for obtaining D-optimal designs for 
specific models using mathematical programming methods (Mitchell, 1974, Craig, 1978). 
There are also numerous software packages available. 

In this study, JMP ® software was utilized to construct the D-Optimal design matrix 
given in Table 2. With this design matrix, the four parameters are studied at three levels 
(values) as represented in coded form by, -1, 0 and +1. As an example, a -1 for fineness 
ratio corresponds to 4 (lower bound), a 0 corresponds to 5.5 (mid value) and +1 
corresponds to 7 (upper bound). These coded values are then transformed into actual 
parameter values to be used in conducting the analysis. The following noise parameter 
matrix. 

5.4. Conduct the Matrix Experiments 
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In this study, all of the geometry and subsystem packaging of the SS V were performed 
using a NASA-developed geometry modeling tool. The Aerodynamic Preliminary 
Analysis System (APAS) was used to determine vehicle aerodynamics. The weights and 
sizing analyses were performed using the NASA-developed Configuration Sizing 
(CONSIZ) weights/sizing package. This process was continued for the 45 rows of the D- 
optimal design matrix, each of which corresponds to a vehicle design generating the data. 

For each of the 45 rows, experiments were repeated at 5 points corresponding to the five- 
point design matrix shown below to simulate the variability (Phadke, 1989) due to the 
uncertain values of the two noise variables (Ballast weight and Mass-Ratio). From the 
weight analysis, data points for empty weight were obtained (Table 2). 


BL 

-1 

+1 

-1 

+1 

0 


-1 

-1 

+1 

+1 

0 



FR 

WA 

TP 

BF 

Weightl 

Weight 2 

Weight 3 

Weight 4 

Weights 

1 

-1 

-1 

■El 

-1 

234798 

266104 

269950 

309903 

268058 

2 
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-1 
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7 

-1 

-1 

1 
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10 
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-1 
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0 
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0 

1 

-1 
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13 
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0 

1 

0 
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407264 

480087 

403068 

14 
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1 

1 

1 

-1 
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44 

1 

1 

1 

1 
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281946 
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45 

1 

1 

1 

1 
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Table 2. Analysis Results 


Using this data, mean empty weights and corresponding standard deviations (as a 
measure of design risk due to the uncertain values of BL and MR) were computed (Table 

3 ). 



FR 

WA 

TP 

BF 
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Std. Devn . 
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-1 
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21 
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43 

1 
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44 

1 

1 

1 

i 

281273 
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45 

1 

1 

1 

i 

298578 
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Table 3. Summary Results for Mean Weight and Standard Deviations 


5.5. Construct the Second-Order Response Surface Model 


Least squares regression analysis is then used to determine the coefficients of the second 
order, dual response surface models for both the mean weight and the standard deviation. 
Table 4 shows the second order response surface model for the standard deviation. 

Y a = c 0 + ? ci xj + ? cy xi 2 + ? ? Cjj xj xj + s (8) 


Term 

Estimate 

Intercept 

28642.707 

FR 

-9513.122 

WA 

6827.781 

TFA 

3643.3254 

BFL 

542.11413 

FR*FR 

4753.1833 

WA*FR 

-3272.696 

WA*WA 

267.37427 

TFA*FR 

-1614.534 

BFL*FR 

-301.9769 

BFL*WA 

-207.5474 

BFL*TFA 

-91.53414 

BFL*BFL 

-450.2649 


Table 4. Standard Deviation Response Surface Model 


R-Sauare 

0.9976 

Adjusted R-Square 

0.9965 

RMS Error 

3961.58 


Table 5. Standard Deviation Model Regression Statistics 

Table 5 displays regression analysis results. The model fit is very good in this case with an 
indicated Adjusted R-square value of 0.9965. 


Table 6 shows the second order response surface model for the mean weight and model 
coefficients. Table 7 displays regression analysis results for the mean weight model. The 
model fit is also very good in this case with an indicated Adjusted R-Square value of 
0.9944. 


Y, = b 0 + ? bi xj + ? bji xj ^ + ? ? bjj xj xj + e 


( 9 ) 


Term 

Estimate 

Intercept 

271645.78 
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FR 

-66299.86 

WA 

43696.134 

TFA 

24725.039 

BFL 

5095.908 

FR*FR 

34505.503 

WA*FR 

-17905.27 

TFA*FR 

-8992.935 

BFL*FR 

-723.6003 

BFL*TFA 

506.43229 

BFL*BFL 

-1715.667 


Table 6. Mean Weight Response Surface Model 


R-Square 

0.9962 

Adjusted R-Square 

0.9944 

RMS Error 

30857.2 


Table 7. Mean Weight Model Regression Statistics 


These dual response surface equations can be used to rapidly determine the effect of 
varying design parameter values on the (weights) performance characteristics and for 
MDO. 

5.6. Determine Design Parameter Values that Optimize the Response 

The purpose in this study is to determine the optimum values of the design parameters to 
considering both the mean empty weight and variance. To obtain a robust design, one 
would seek to minimize the variability (risk) subject to a constraint on weight. Therefore, 
the optimization problem was set up as follows: 

Minimize Y„ 


Subject to: 

Yu = 331704 

-1 = FR = 1 
-1 = WA = 1 
-1 = TP = 1 
-1 = BF = 1 

This optimization problem was solved by utilizing three approaches; Excel Solver, GINO 
and Genetic Algorithms. Solver, available in Microsoft Excel®, is a gradient based 
optimization algorithm. GINO also uses a gradient based optimization algorithm and is 
available as an independent optimization software package (Liebman et al, 1986). Genetic 
Algorithms mimic natural search and selection processes leading to the survival of fittest 
individuals. In the range of optimization techniques, GA occupy a gap between calculus 
based methods and random search (Goldberg, 1989). They are in general less efficient than 
gradient based methods, however they can be applied to multi-modal problems with 
discontinuous topologies (Gage, 1994). The GA results in this study were obtained from 


9 





Genetik (floating point GA for minimization problems. Genetik V2.01 is available at: 
http://www.umoncton.ca/turk/genetik201.zip, for downloading. 

Table 8 summarizes the optimization results. Solver and the Genetic Algorithm has found 
identical results, which is better than the GINO results in this case. Since Solver and GA 
are both utilized at VAB in prior studies, the results support their continued use in this type 
of optimization problems where the response surfaces usually have been found to have a 
saddle shape. 


1 Excel Solver 

GINO 

I Genetic Algorithm ! 

Parameter 

Optimized 

Value 

Parameter 

Optimized 

Value 

Parameter 

Optimized 

Value 

FR 

0.455864 

FR 

0.518376 

FR 

0.4548447 

WA 

-i 

WA 

-1 

WA 

-i 

TP 

-1 

TP 

-1 

TP 

-i 

BF 

-1 

BF 

-1 

BF 

■1 







Std.Devn 

16164 

Std.Devn 

17552.66 

Std.Devn 

16164.16 

Mean Wt 

186465 

Mean Wt 

194570.55 

Mean Wt 

185465.48 


Table 8. Optimization Results 


Since the goal in this study was to search for optimization results considering both the 
mean and the variability, a mean and variance graph is plotted below. Table 9 indicates 
that both the mean and the variability are minimized at x'=(0.454844;-l;-l;-l) for this 
problem. Even though this is a desirable result, one would expect that for many 
optimization problems, there would be a trade off between the mean and variance. 
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Mean vs Std. Devn. 

16220 
16210 
c 16200 

Q 16190 

2 

(0 16180 
16170 
16160 

186000 186200 186400 186600 186800 187000 187200 

Mean 



Mean 

Std. Devn. 

187000 

16180 

186900 

16175 

186800 

16171 
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Table 9: Mean versus Standard Deviation Plot 


6. Conclusions 

Space transportation system conceptual design is a multidisciplinary process containing 
considerable element of uncertainty. Uncertainties of one subsystem (and/or discipline) 
may propagate to another, through linking parameters and the final system output has an 
accumulation of risk. Risk here is defined as the variability in the expected system output 
performance characteristics. 

This study investigated the use of dual responses to estimate variability resulting from 
uncertain values of several design parameters (called the noise factors) and conduct MDO 
considering both the mean and variance. The dual response surface approach is applied 
to the conceptual design of a wing-body vehicle, configuration study. 


11 




An orthogonal array based simulation approach is used to simulate the variability 
resulting from noise factors. With this approach, generation of the additional data points 
needed to construct a second response surface for the standard deviation was not costly in 
this case since the analysis was already set up to run the design points for constructing the 
response surface for the mean weight. The results here indicate that in many cases the 
second response surface can be constructed without much extra effort. Once this is done, 
a measure of design risk is obtained and optimization on risk basis can be conducted. 

The results therefore suggest that the dual response surface approach can be utilized in 
approximation model building to estimate the variability in the mean value of the 
performance characteristic in conceptual design. Subsequently, multidisciplinary 
optimization studies can be conducted considering both the mean response and variability 
to reduce design risk and seek a robust design solution. 
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8. Appendix A 

In this study, an estimate of population mean and variance for empty-weight were 
computed based on the experimental results. Given the mean and variance, one could 
then compute a probability associated with a given weight value and construct a 
cumulative distribution function. In order to be able to do that, one needs to make an 
assumption on the underlying population distribution. The most commonly made 
assumption in many studies is the “Normal Distribution” assumption. However, if this 
assumption is not correct, the results and predictions made on probability will be 
misleading. 

To test such an assumption a statistical test of normality was conducted. As the following 
results indicate, the weights data clearly is not normally distributed. Hence results based 
on such an assumption would be incorrect. A Kolmorgorov-Smimov (KS) hypothesis 
test was conducted to test for normality of the underlying population distribution. 

Ho: there is no difference between the distribution of the data set and a normal one 
H a : there is a difference between the distribution of the data set and normal 

Test of Normality 

Statistic Degrees of Freedom Significance 

0.086 225 0.0003692 

The Kolmorgorov-Smimov statistic is significant. Hence the normality assumption is not 
satisfied (p<0.05). The hypothesis of normality is rejected (The distribution of data is 
significantly different from normal). 
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Q-Q plot is a plot of our observed values against the expected values. If the data is from 
a normal distribution, the data points should fall on a straight line. In the Figure above , 
the expected normal distribution is the straight line and the line of little boxes is the 
observed values from our data. So, the distribution of data is not normal. 
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